> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/Anny26022/chartsmaze_clone/llms.txt
> Use this file to discover all available pages before exploring further.

# Pipeline Architecture

> Understanding the 6-phase design and dependency structure of the EDL Pipeline

The EDL Pipeline follows a **strict 6-phase architecture** designed to orchestrate 16+ data fetching and processing scripts in the correct dependency order. This page explains why order matters, how phases depend on each other, and the configuration options available.

## Architecture Overview

The pipeline is orchestrated by `run_full_pipeline.py`, which executes scripts sequentially across six phases:

```mermaid theme={null}
graph TD
    A[PHASE 1: Core Data] --> B[PHASE 2: Data Enrichment]
    B --> C[PHASE 2.5: OHLCV Data]
    C --> D[PHASE 3: Base Analysis]
    D --> E[PHASE 4: Enrichment Injection]
    E --> F[PHASE 5: Compression]
    F --> G[PHASE 6: Optional Standalone]
    
    style A fill:#e1f5ff
    style D fill:#fff4e1
    style E fill:#e8f5e9
    style F fill:#f3e5f5
```

## Phase Breakdown

### PHASE 1: Core Data (Foundation)

Creates the foundational datasets that all other scripts depend on.

<Accordion title="Scripts & Outputs">
  | Script                      | Output                                                | Purpose                                       |
  | --------------------------- | ----------------------------------------------------- | --------------------------------------------- |
  | `fetch_dhan_data.py`        | `dhan_data_response.json`<br />`master_isin_map.json` | Fetches 2,775 stocks and creates ISIN mapping |
  | `fetch_fundamental_data.py` | `fundamental_data.json`                               | Quarterly results & financial ratios (35 MB)  |
  | NSE CSV Download            | `nse_equity_list.csv`                                 | Listing dates for all stocks                  |
</Accordion>

<Note>
  **Critical Dependency**: `master_isin_map.json` is used by **ALL** scripts in Phase 2, 2.5, and 4. If `fetch_dhan_data.py` fails, the pipeline cannot continue.
</Note>

### PHASE 2: Data Enrichment (Fetching)

Parallel execution of 11 data fetching scripts, all consuming `master_isin_map.json`.

<Accordion title="Scripts & Outputs">
  | Script                             | Output                                                                  | Description                                                 |
  | ---------------------------------- | ----------------------------------------------------------------------- | ----------------------------------------------------------- |
  | `fetch_company_filings.py`         | `company_filings/{SYMBOL}_filings.json`                                 | Hybrid LODR + Legacy filings                                |
  | `fetch_new_announcements.py`       | `all_company_announcements.json`                                        | Live corporate announcements                                |
  | `fetch_advanced_indicators.py`     | `advanced_indicator_data.json`                                          | Pivot Points, EMA/SMA signals (8.3 MB)                      |
  | `fetch_market_news.py`             | `market_news/{SYMBOL}_news.json`                                        | AI-sentiment news (50/stock)                                |
  | `fetch_corporate_actions.py`       | `upcoming_corporate_actions.json`<br />`history_corporate_actions.json` | Dividends, Bonus, Splits (2 years history + 2 months ahead) |
  | `fetch_surveillance_lists.py`      | `nse_asm_list.json`<br />`nse_gsm_list.json`                            | ASM/GSM surveillance lists                                  |
  | `fetch_circuit_stocks.py`          | `upper_circuit_stocks.json`<br />`lower_circuit_stocks.json`            | Circuit breaker stocks                                      |
  | `fetch_bulk_block_deals.py`        | `bulk_block_deals.json`                                                 | Bulk/Block deals (30 days)                                  |
  | `fetch_incremental_price_bands.py` | `incremental_price_bands.json`                                          | Daily price band changes                                    |
  | `fetch_complete_price_bands.py`    | `complete_price_bands.json`                                             | All securities price bands                                  |
  | `fetch_all_indices.py`             | `all_indices_list.json`                                                 | 194 market indices                                          |
</Accordion>

### PHASE 2.5: OHLCV Data (Smart Incremental)

Optional phase controlled by `FETCH_OHLCV` flag. Downloads lifetime historical OHLCV data with intelligent incremental updates.

<Accordion title="Scripts & Performance">
  | Script                   | Output                           | Performance                                |
  | ------------------------ | -------------------------------- | ------------------------------------------ |
  | `fetch_all_ohlcv.py`     | `ohlcv_data/{SYMBOL}.csv`        | \~2-5 min incremental, \~30 min first-time |
  | `fetch_indices_ohlcv.py` | `ohlcv_data/indices/{INDEX}.csv` | High-speed specialized fetcher             |

  **Incremental Logic**: Only downloads missing dates if CSV exists, full history otherwise.
</Accordion>

<Warning>
  If `FETCH_OHLCV = False`, the following fields will be zero in the final output:

  * ADR (Average Daily Range)
  * RVOL (Relative Volume)
  * ATH & % from ATH
  * All turnover metrics
  * Post-earnings returns
</Warning>

### PHASE 3: Base Analysis (Building Master JSON)

Single critical script that produces the base structure of `all_stocks_fundamental_analysis.json`.

<Accordion title="bulk_market_analyzer.py Details">
  **Inputs**:

  * `fundamental_data.json` (Phase 1)
  * `dhan_data_response.json` (Phase 1)
  * `advanced_indicator_data.json` (Phase 2)
  * `nse_equity_list.csv` (Phase 1)

  **Outputs**:

  * `all_stocks_fundamental_analysis.json` (Base structure with \~60 fields)

  **Processing**:

  1. Loads fundamental data for all 2,775 stocks
  2. Merges technical data from Dhan response
  3. Adds advanced indicators (Pivots, SMA/EMA status)
  4. Calculates QoQ/YoY growth metrics
  5. Computes valuation ratios (P/E, PEG, ROE, ROCE, D/E)
  6. Adds shareholding patterns (FII/DII changes, Free Float)
</Accordion>

<Note>
  This script MUST complete successfully before Phase 4. All Phase 4 scripts modify this JSON file **in-place**.
</Note>

### PHASE 4: Enrichment Injection (Order Matters!)

Five scripts that sequentially inject additional fields into `all_stocks_fundamental_analysis.json`.

<Warning>
  **CRITICAL**: These scripts MUST run in this exact order. Each modifies the JSON file in-place.
</Warning>

<Tabs>
  <Tab title="Order & Dependencies">
    ```mermaid theme={null}
    graph LR
        A[advanced_metrics_processor.py] --> B[process_earnings_performance.py]
        B --> C[enrich_fno_data.py]
        C --> D[process_market_breadth.py]
        D --> E[process_historical_market_breadth.py]
        E --> F[add_corporate_events.py]
        
        style F fill:#ffebee
    ```

    | Order | Script                                 | Fields Added                                                               | Dependencies                      |
    | ----- | -------------------------------------- | -------------------------------------------------------------------------- | --------------------------------- |
    | 1     | `advanced_metrics_processor.py`        | ADR, RVOL, ATH, Turnover, Gap Up %, Day Range %                            | `ohlcv_data/`                     |
    | 2     | `process_earnings_performance.py`      | Quarterly Results Date, Returns since Earnings, Max Returns since Earnings | `company_filings/`, `ohlcv_data/` |
    | 3     | `enrich_fno_data.py`                   | F\&O Flag, Lot Size, Next Expiry                                           | F\&O data fetchers                |
    | 4     | `process_market_breadth.py`            | Relative Strength Rating, Market Breadth metrics                           | Returns data from base analysis   |
    | 5     | `process_historical_market_breadth.py` | Historical breadth charts                                                  | OHLCV data                        |
    | 6     | `add_corporate_events.py`              | Event Markers, Recent Announcements, News Feed                             | ALL Phase 2 outputs               |
  </Tab>

  <Tab title="Why Order Matters">
    ### Dependency Chain Explained

    **1. Advanced Metrics First**

    * Calculates volatility (ADR) and volume (RVOL) from OHLCV data
    * Must run before earnings processor (which needs price gap calculations)

    **2. Earnings Performance Second**

    * Reads company filings to find quarterly results dates
    * Calculates returns from earnings date using OHLCV data
    * Requires base price data to be present

    **3. F\&O Data Third**

    * Independent enrichment, no dependencies on previous steps
    * Can technically run earlier, but positioned here for logical grouping

    **4. Market Breadth Fourth**

    * Needs return calculations from base analysis
    * Computes relative strength vs. market indices

    **5. Corporate Events LAST**

    Aggregates data from ALL Phase 2 sources:

    * `company_filings/` → Recent Announcements
    * `market_news/` → News Feed
    * `upcoming_corporate_actions.json` → Event Markers
    * `nse_asm_list.json` → Surveillance flags
    * `bulk_block_deals.json` → Block deal markers
    * `incremental_price_bands.json` → Circuit revision markers

    Must be final because it creates summary fields from all other data.
  </Tab>
</Tabs>

### PHASE 5: Compression

Compresses final outputs to `.json.gz` format with maximum compression.

<Accordion title="Compression Details">
  **Files Compressed**:

  * `all_stocks_fundamental_analysis.json` → `.json.gz` (\~80% smaller)
  * `sector_analytics.json` → `.json.gz`
  * `market_breadth.csv` → `.json.gz`

  **Typical Results**:

  * Raw JSON: \~35-40 MB
  * Compressed: \~7-8 MB
  * Compression ratio: 80%+
</Accordion>

### PHASE 6: Optional Standalone Data

Controlled by `FETCH_OPTIONAL` flag. Produces standalone datasets **not included** in the master JSON.

<Accordion title="Optional Scripts">
  | Script                 | Output                   | Description        |
  | ---------------------- | ------------------------ | ------------------ |
  | `fetch_all_indices.py` | `all_indices_list.json`  | 194 market indices |
  | `fetch_etf_data.py`    | `etf_data_response.json` | 361 ETF details    |

  **Note**: These are standalone products and not consumed by the master pipeline.
</Accordion>

## Configuration Flags

Edit these flags inside `run_full_pipeline.py` (lines 60-71):

<CodeGroup>
  ```python FETCH_OHLCV theme={null}
  # OHLCV: Auto-detect mode
  # True = always fetch (incremental update: ~2-5 min if data exists, ~30 min first time)
  # False = skip entirely (ADR, RVOL, ATH, % from ATH fields will be 0)
  FETCH_OHLCV = True
  ```

  ```python FETCH_OPTIONAL theme={null}
  # Set to True to also fetch standalone data (Indices, ETFs)
  FETCH_OPTIONAL = False
  ```

  ```python CLEANUP_INTERMEDIATE theme={null}
  # Auto-delete intermediate files after pipeline succeeds
  # Keeps: all_stocks_fundamental_analysis.json.gz + ohlcv_data/
  CLEANUP_INTERMEDIATE = True
  ```
</CodeGroup>

### Impact of Configuration

<Tabs>
  <Tab title="FETCH_OHLCV">
    | Setting | Impact                                    | Runtime                | Output Fields           |
    | ------- | ----------------------------------------- | ---------------------- | ----------------------- |
    | `True`  | Full OHLCV download + incremental updates | +2-30 min              | All 86 fields populated |
    | `False` | Skip OHLCV entirely                       | Faster (\~4 min total) | 15+ fields will be zero |

    **Zero Fields when False**:

    * ADR (5/14/20/30 Days MA)
    * RVOL
    * ATH, % from ATH
    * Gap Up %, Day Range %
    * % from 52W Low
    * 6 Month Returns
    * 200 Days EMA Volume
    * Daily Rupee Turnover (20/50/100)
    * 30 Days Average Rupee Volume
    * Returns since Earnings
    * Max Returns since Earnings
  </Tab>

  <Tab title="CLEANUP_INTERMEDIATE">
    | Setting | Disk Usage After Pipeline | Files Kept                       |
    | ------- | ------------------------- | -------------------------------- |
    | `True`  | \~50-100 MB               | `.json.gz` + `ohlcv_data/` only  |
    | `False` | \~200-300 MB              | All intermediate JSONs + outputs |

    **Deleted Files** (when True):

    * `master_isin_map.json`
    * `dhan_data_response.json`
    * `fundamental_data.json` (35 MB)
    * `advanced_indicator_data.json` (8.3 MB)
    * `all_company_announcements.json`
    * `company_filings/` directory
    * `market_news/` directory
    * All corporate action JSONs
    * Raw `all_stocks_fundamental_analysis.json` (kept as `.gz` only)
  </Tab>
</Tabs>

## Error Handling Strategy

The pipeline implements a **resilient continuation strategy**:

<Steps>
  <Step title="Critical Failures">
    If `fetch_dhan_data.py` (Phase 1) or `bulk_market_analyzer.py` (Phase 3) fail, the pipeline **stops immediately**.

    These scripts produce the master ISIN map and base JSON that all other scripts depend on.
  </Step>

  <Step title="Enrichment Failures">
    If any Phase 2 or Phase 4 script fails, the pipeline **continues** and marks the script as failed.

    This ensures you get a complete output even if individual data sources are temporarily unavailable.
  </Step>

  <Step title="Final Report">
    At completion, the pipeline reports:

    * Total runtime
    * Successful scripts count
    * Failed scripts list
    * Output file size and compression ratio
  </Step>
</Steps>

## Performance Characteristics

<CardGroup cols={2}>
  <Card title="Minimal Run" icon="bolt">
    **Configuration**: `FETCH_OHLCV = False`

    **Runtime**: \~4 minutes

    **Output**: 60+ fields per stock (missing volume/volatility metrics)
  </Card>

  <Card title="Full Run" icon="database">
    **Configuration**: `FETCH_OHLCV = True` (incremental)

    **Runtime**: \~6-9 minutes

    **Output**: All 86 fields per stock
  </Card>

  <Card title="First-Time Full Run" icon="rocket">
    **Configuration**: `FETCH_OHLCV = True` (no existing data)

    **Runtime**: \~35-40 minutes

    **Output**: All 86 fields + complete OHLCV history
  </Card>

  <Card title="With Cleanup" icon="trash">
    **Configuration**: `CLEANUP_INTERMEDIATE = True`

    **Disk Saved**: \~150-200 MB

    **Retained**: Only `.json.gz` + `ohlcv_data/`
  </Card>
</CardGroup>

## Next Steps

<CardGroup cols={2}>
  <Card title="Data Flow" icon="diagram-project" href="/concepts/data-flow">
    Understand how data transforms across phases
  </Card>

  <Card title="Output Schema" icon="table" href="/concepts/output-schema">
    Explore the 86 fields in the final JSON
  </Card>

  <Card title="Quick Start" icon="play" href="/quickstart">
    Run your first pipeline
  </Card>

  <Card title="Configuration" icon="sliders" href="/configuration/pipeline-settings">
    Customize pipeline behavior
  </Card>
</CardGroup>